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Abstract 

The purpose of this thesis was to review current Systems Program Office (SPO) 
effectiveness measures in light of the recently implemented Government Performance and 
Results Act of 1993 (GPRA), and to develop an integrated model of organizational 
effectiveness SPOs can use to comply with GPRA. Specifically, this thesis attempted to 
answer three research propositions addressing whether or not: 

1. Current effectiveness measures used within SPOs are strictly output goal measures; 

2. GPRA, while emphasizing the need for a more “holistic” approach has not provided 
a good model to accomplish this; and 

3. An integrated model can be developed to meet both the spirit of GPRA and the 
SPOs’ need for a thorough mechanism for measuring organizational effectiveness. 

These research propositions were answered through a comprehensive literature review, 
personal interviews conducted with SPO personnel, and SPO records analysis. The 
research identified the current SPO effectiveness measures do not comply with GPRA’s 
intent to focus attention on strategic planning and outcomes. Rather, SPO effectiveness 
measures focus almost exclusively on output measures which tend to be reactive 
measures. Strictly output measures indicate a lack of emphasis on outcomes which are 
mandated by GPRA and defined as measures of effectiveness from a customer satisfaction 
perspective. Further, research yielded an integrated model which can help SPOs 
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implement GPRA and comply with its mandated emphasis on strategic planning and 
related outcome measures of effectiveness. 
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DEVELOPING AN INTEGRATED MODEL FOR MEASURING 


ORGANIZATIONAL EFFECTIVENESS IN LIGHT OF THE GOVERNMENT 
PERFORMANCE AND RESULTS ACT OF 1993 


1. Introduction 

In the early 1970’s, Schlitz Brewing Company was the model organization in the 
brewing industry. By the late 1970’s, Schlitz had fallen to number seven in the industry 
(Dang, 1993:4). Volume sales had plummeted by over 40%, company stock had lost $1.7 
billion in value, and independent wholesalers, disappointed with the company, turned to 
other brands for business (Dang, 1993:4). Schlitz’s deterioration is a classic example of 
what can happen when an organization loses its strategic focus. A review of the data 
suggests that Schlitz’s failure can be attributed to the company’s preoccupation with 
short-term goals. For example, Schlitz’s failure to diversify and a fixation on immediate 
cost reduction, both short-term perspectives, ultimately resulted in its free fall from the top 
of the industry. Dang illustrates the delicate balance that exists between short and long¬ 
term goals by writing: 

Which time perspective is most appropriate in assessment? For 
example, if current production (a short-run criterion) consumes 
so much of an organization’s resources that little is left over for 
investment in R&D, the organization may ultimately find itself 
with its products outmoded and its very survival (a long-term 
criterion) threatened. (Dang, 1993:21) 
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One could argue that an even more basic reason for the failure was Schlitz’s inability to 
relate organizational effectiveness with strategic management. Gibson writes that 
organizations are “entities that enable society to pursue accomplishments that can’t be 
achieved by individuals acting alone” (Gibson and others, 1994:5). Said another way, 
organizations form as a result of individuals deliberately seeking to take advantage of 
synergy, the idea that the end product is greater than the sum of its constituent parts. 
However, to obtain and sustain effectiveness, organizations must have goals that it strives 
to achieve. These goals must be periodically reviewed and updated to ensure the 
organization has something to work towards. Richard Steers captures this belief by 
stating that “the concept of organizational effectiveness is best understood in terms of a 
continuous process rather than an end state” (Steers, 1976:32). Organizations strive to be 
effective for many reasons. Effectiveness translates into increased profit, growth, market 
share, and ultimately, continued existence. Examples such as Schlitz have placed a 
premium on understanding the relationship between organizational effectiveness and 
accomplishing organizational goals. 

As the Air Force continues to fight over ever dwindling Congressional appropriations. 
Air Force managers must place a strong emphasis on understanding the relationship 
between organizational effectiveness and goal accomplishment. Much like Schlitz, the Air 
Force is operating in a very dynamic environment characterized by downsizing, changing 
roles, and changing missions. To avoid a performance like Schlitz’s, the Air Force must 
avoid the tendency for organizations to become preoccupied with short-term efficiency 
when faced with a dynamic environment. That is precisely why the Government 
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Performance and Results Act of 1993 (GPRA) can be so useful to Air Force managers. 
GPRA, if successful, will form the link between organizational effectiveness and strategic 
planning. Such a link will help ensure that the Air Force does not lose its strategic focus 
in an environment which severely punishes shortsightedness. 

In recent years, the Department of Defense (DoD) has come under close scrutiny by 

Congress and the American public for its handling of high dollar, major weapon system 

acquisitions. For example, in 1991, then Secretary of Defense Dick Cheney, under 

pressure from Congress, canceled the Navy’s A-12 aircraft acquisition citing gross 

mismanagement and unacceptable cost overrans. Government acquisitions have become 

characterized by terms such as “over budget” (Nicholas, 1990:341). Nicholas writes, 

“Cutting edge, high-technology and R&D projects frequently show cost escalation 

upwards of several hundred percent...and NASA spacecraft often exceed estimates by a 

factor of four to five” (Nicholas, 1990:342). Recognizing that public trust and efficiency 

must be injected into DoD business. Congress passed GPRA. In the preamble to the Act, 

Congress unequivocally stated its reasons and purpose in passing the Act: 

Public confidence in the institutions of American government is 
suffering from a perception that those institutions are not working 
well...can be seen in the results of a recent public opinion poll 
which shows that Americans, on average, believe that as much 
as 48 cents out of every Federal tax dollar is wasted. In other 
words, the public believes that it is not getting the level and quality 
of government service for which it is paying...The Committee shares 
in the public’s frustration with waste, inefficiency, and ineffectiveness 
...the Committee believes that the regular and systematic measurement 
and reporting of program performance, compared to pre-established 
goals, would be a major addition...Govemmental waste and under¬ 
performance will likely persist until fhere is a change in the behavior 
of federal agencies...The legislation will provide the information 
necessary to strengthen program measurement, to make objective 


3 





evaluations of program performance, and to set realistic, measurable 
goals for future performance. (United States Senate, 1993:2-3) 

The vehicle by which Congress will attempt to meet these goals of efficiency and 
increased effectiveness is strategic planning. How Congress will measure organizational 
effectiveness remains to be seen. Traditional measures of organizational effectiveness 
have been output goal models. For example, in measuring Systems Program Office (SPO) 
effectiveness. Congress has monitored cost control and schedule maintenance, both of 
which are end results of the systems acquisition process. Such measures, in many cases, 
neglect other important factors critical to the effective performance of any organization. 
For example, information on stakeholder satisfaction and the quality of the transformation 
process are lost when the focus of measurement is strictly output oriented. 

The intent of GPRA “is to improve the efficiency and effectiveness of Federal 

programs by establishing a system to set goals for program performance and to measure 

results (United States Senate, 1993:2). However, the General Accounting Office (GAO), 

a longtime advocate of such a system, intends to use GPRA as a means of allocating the 

federal budget. Donald H. Chapin, Assistant Comptroller General for Accounting and 

Financial Management, states: 

..if we can get the agencies to supply us with measurable 
goals, we can relate those to financial results and then you 
can see what you are spending your money on and whether 
that money is well spent. And that is my fond hope, that we 
can get that into our system of Government and have it 
reported to the Committees of Congress as a regular matter 
so that they can see the effectiveness of the money that they 
appropriate. (United States Senate, 1993:4-5) 
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Such a statement indicates how critical it will be for Government agencies to correctly 
implement strategic planning and accurately measure organizational effectiveness. 

In recent studies of organizations, researchers have focused on developing models for 
measuring organizational effectiveness. In general, these models can be classified into 
four categories: output goal, multiple constituency, systems resource, and the process 
approach (Cameron, 1980:67). 

Overview of Thesis 

The purpose of this thesis is to review current SPO effectiveness measures in light of 
GPRA,“and to develop an integrated model of organizational effectiveness SPOs can use 
to comply with GPRA’s intent. As will be discussed, the main strength of the integrated 
model is its ability to incorporate GPRA’s main purpose, namely to serve as the link 
between measures of effectiveness and strategic planning. Additionally, the model uses a 
systems approach which stresses the importance of stakeholder feedback and de- 
emphasizes the pursuit of strictly short-term, output measures of effectiveness. 

Chapter II will provide some general background information on organizational 
effectiveness to include a brief discussion of the four traditional models mentioned earlier. 
The chapter will be concluded with a discussion of both GPRA and the integrated model 
that will be used to assess SPO effectiveness measures. 

Chapter III will present the methodology to be employed during the study. This 
methodology will be a qualitative methodology that uses triangulation theory as its 
foundation. 
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Chapter IV will present the data gathered via the qualitative methodology, and will 
close with a depiction of the model showing how SPOs are currently operating. 

Chapter V will analyze the data presented in Chapter IV, and will discuss what SPOs 
can do to improve their effectiveness measurement process. The integrated model will be 
used to analyze the data and to draw conclusions from the data. 

Definitions 

Several key terms and definitions relevant to this thesis are provided below: 

1. Output measure - the amount of product or service produced by an organization’s 
transformation process. Outputs are the result of the transformation process and typically 
are not measured from the customer’s perspective. Therefore, it is common for output 
measures to be associated with process efficiency. 

2. Outcome measure — an assessment of the results of a program activity or product 
compared to its intended purpose. It is different from an output measure which merely 
counts the amount of product or service produced by the transformation process. Rather, 
outcomes flow from products and are viewed from a customer perspective, not a process 
perspective. As such, discussions of outcomes and effectiveness are intertwined as are 
discussions of outputs and efficiency. 

3. System Program Office (SPO~) -- a collection of personnel and resources from various 
functional disciplines working together to field a system or capability which fulfills a 
validated customer need. 
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4. Metric - a measurement over time, that communicates vital information about a 
process or activity, drives appropriate actions, and is linked to a strategic plan (AFMC 
Metrics Course, 1994:47). 

5. Effectiveness - an aspect of performance that describes how well the organization’s 
activities are contributing to achieving intended outcomes. Effectiveness is a 
“consequence” measure. In terms of GPRA, effectiveness means the consequence of an 
output from the customer’s perspective. 

6. Efficiency -- an aspect of performance that describes the relationship between inputs 
and outputs. Efficiency is measured from a process perspective and not a customer 
perspective as is done with effectiveness. 
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n. Literature Review 


Overview 

This chapter provides a background for the research topic of developing an integrated 
model for measuring organizational effectiveness. The chapter will begin with an 
introduction of why organizational effectiveness is important and how Congress intends to 
ensure it. The introduction will be followed by a definition of organizational effectiveness 
developed from an illustration of four traditional models used to measure it. Next will be 
a brief analysis of the four models and a discussion of the current trend in organizational 
effectiveness literature. The chapter will be closed with a review of GPRA, and how it fits 
into the current stream of effectiveness literature. Based on this discussion, a series of 
research questions to be addressed by this study will be presented. 

Theoretical Background 

Although many models of organizational effectiveness exist, most of them are 
grounded in four traditional models. These models include: output goal, multiple 
constituency, systems resource, and the process approach model (Cameron, 1980:67). 

Output Goal Model . The first and most widely used approach defines effectiveness in 
terms of how well an organization accomplishes its goals—the closer the organization’s 
outputs come to meeting its goals, the more effective it is (Cameron, 1990:67; Gibson and 
others, 1994:31). For example, a business may set an annual sales goal of $10 million or a 
production goal of 1000 units per month. If the firm exceeds $10 million in sales or 
produces units at a rate that exceeds 1000 units per month, the firm would be considered 
effective. 
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Multiple Constituency Model. This model has also been termed the strategic 
constituencies approach, the participant satisfaction model, and the stakeholder 
management model (Cameron, 1980:67; Sturdivant, 1979:54). Sturdivant explains the 
concept by defining stakeholder groups as “people who are affected by corporate policies 
and practices and who see themselves as having a stake in the business” and “any group 
whose collective behavior can directly affect the organization’s future, but which is not 
under the organization’s direct control” (Sturdivant, 1979:54). For example, stakeholders 
in the DoD would include politicians and taxpayers. Stakeholder management attempts to 
achieve an equitable balance between the objectives of the corporation and the interests of 
the stakeholder groups (Sturdivant, 1979:54). An effective firm is one that is able to meet 
its objectives while satisfying the interests of stakeholder groups (Zammuto, 1984:606). 

System Resource Model . The focus in this approach to measuring organizational 
effectiveness lies in the ability to obtain resources. Cameron writes: “..an organization’s 
effectiveness is judged on the extent to which it acquires needed resources—that is, the 
more of the needed resources an organization can obtain from its external environment, 
the more effective it is” (Cameron, 1980:67). Here, the emphasis on organizational inputs 
is analogous to the emphasis on outputs in the output goal model. 

Process Approach Model . Cameron writes that: 

...effective organizations are those with an absence of internal 
strain, whose systems are highly integrated into the system, 
whose internal functioning is smooth and typified by trust and 
benevolence toward individuals, where information flows smoothly 
both vertically and horizontally, and so on. (Cameron, 1980:67) 
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Here, organizations are more effective if they are able to achieve these internal 
characteristics and less effective if they cannot. 

Organizational Effectiveness: Analysis 

In analyzing the different organizational effectiveness models, authors agree that no 
model is appropriate in. all circumstances or for all organizational types (Cameron, 

1980:70; Hitt and Middlemist, 1979:373; Ostroff and Schmitt, 1993:1345). Chakravarthy, 
in describing the exclusive use of output goal models, states “they assume that a single 
performance criterion can assess ‘excellence’, they focus only on customers to the 
exclusion of the transformation process within the firm, and they ignore the claims of other 

/ 

stakeholders...” (Chakravarthy, 1986:445). Also critical of exclusive use of the output 
goal model were Hansen and Wemerfelt. They add that both economic factors (i.e. 
output goals) and organizational factors (i.e. process approach measures) are important, 
but organizational factors explain more of the variance in performance than do economic 
factors (Hansen and Wemerfelt, 1989:406). Researchers are equally as critical of the 
exclusive use of the multiple constituency, systems resource, and process approaches 
(Cameron, 1980:68-70; Seashore, 1983:61-62). 

In recent years, however, researchers have begun to push their studies of 

organizational effectiveness models in a new direction. This new direction has been to 

develop hybrid models that heavily weigh stakeholder satisfaction in defining effectiveness. 

For example, Chakravarthy writes: 

This paper demonstrates the inadequacy of traditional measures 
that are based on a firm’s profitability, for evaluating its strategic 
performance. Two other measures, one that attempts to assess the 
quality of a firm’s transformations (and not merely its outcomes) 


10 





and the other that attempts to measure the satisfaction of all of the 
firm’s stakeholders (and not merely it stockholders), are shown 
here to be important discriminators of strategic performance. 
(Chakravarthy, 1986:437) 

Cameron, speaking of the four general models, adds that “each approach is analytically 
independent and that weaknesses exist in all of them” and “researchers have been trying to 
define and evaluate organizational effectiveness to get a framework that will attain general 
consensus” (Cameron, 1980:79). Additionally, Hansen and Wemerfelt integrated two 
sample models of organizational effectiveness, one from the economic viewpoint (i.e. the 
ouQiut goal approach) and one from the organizational viewpoint (i.e. the process 
approach) concluding “the results confirm the importance and independence of both sets 
of factors in explaining performance” (Hansen and Wemerfelt, 1989:406). Lastly, 
Seashore discusses an integrated model in which constituents serve as “integrators” and 
defines effectiveness within the context of the integrated model as “whatever some 
constituent or some researcher making attributions to a constituency says it is” (Seashore, 
1983:65). Given the almost universal belief among researchers that no single model alone 
can fit all situations and organizations, integrated models will take on more importance. 

Evolution of Performance Measures Within SPOs 

The prominent and nearly exclusive performance measurement instrument used by 
SPOs to measure organizational effectiveness is metrics, which can be defined as a 
measurement over time, that communicates vital information about a process or activity, 
drives appropriate actions, and is linked to a strategic plan (AFMC Metrics Course, 
1994:47). Many believe that measurement is a fundamental part of good management. 


11 




and that metrics are merely the way to achieve meaningful measurement. Bill Hewlett, co¬ 
founder of Hewlett-Packard, states “You cannot manage what you cannot measure,” and 
“What gets measured gets done” (House and Price, 1991:93). 

Metrics used by SPOs have evolved and will continue to evolve over time. Air Force 
Material Command (AFMC) has identified three major external change agents that are 
forcing the current phase of evolution: 1) Dwindling resources, 2) Competitive 
environment, and 3) Government reform. The third change agent. Government reform, is 
the focus of this thesis and will be studied more closely. Two examples of Government 
reform cited by AFMC are GPRA and Executive Order 12862 entitled “Setting Customer 
Service Standards.” GPRA has been mentioned and will be outlined in the next section. 
Executive Order 12862, signed by President Clinton on 11 September 1993, will require 
Government agencies to have customer focus and reward performance and allocate 
resources based on customer satisfaction results (Clinton, 1993: 1737-1738). The 
evolution of SPO performance measures is illustrated in Figure 2.1 below: 
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The Evolution of Performance Measures 
• Functional Perspective -► • Global/System Perspective 



Lack of Customer Focus — 

—► • 

Customer Focus “Designed In” 


Not Strategically Linked — 

—► • 

Strategically Aligned 


Control — 

—► • 

Empowerment 


Management by Emotion— 

—► • 

Management by Facts 


Variation Misinterpreted — 

—► • 

Variation Understood 


Quick Fixes — 

—► • 

Continuous Improvement 


Reactive — 

—► • 

Proactive 


Figure 2.1. Evolution of SPO Performance Measures 
(AFMC Metrics Course, 1994:61) 


From this relationship has emerged the core relationship a good metric must capture. 
AFMC calls this relationship the “customer-product-process relationship” or “CP2” and 
states that all modem measures (metrics) must focus on the entire relationship (outcome). 
A good metric or group of metrics must therefore incorporate the entire system to include 
inputs, processes, outputs, and customers. In other words, a good metric or group of 
metrics must integrate all the players in the system. This relationship is depicted in Figure 
2.2 below: 
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Expanding Our Management Perspective 



Figure 2.2. Customer-Product-Process Relationship 
(AFMC Metrics Course, 1994:60-61) 


The slide above incorporates the concepts of efficiency and effectiveness, the primary 
motivators Congress cited in its preamble to GPRA. Therefore, it can be said that AFMC 
recognizes that GPRA will impact performance measurement (metrics) currently in use. 
Additionally, AFMC recognizes that metrics must be linked to strategic plans (AFMC 
Metrics Course, 1994:10) and must embody the CP2 relationship. This emphasis on 
strategic planning and customers is reflected in the following excerpt from the AFMC 
Metrics Course Handbook : 

The command’s strategic planning process provides the foundation 
for metrics. Strategic plans at every level within the command should 
be aligned with the organizational level above it. It is essential that 
AFMC personnel understand that metrics are not an end to themselves 
but are by-products of effective strategic planning that link 
organizational mission, goals and objectives to their day-to-day work. 

(33) 
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This thesis will investigate, among other things, whether current SPO effectiveness 
measures (metrics) incorporate GPRA concepts. Also, this thesis will address whether 
current effectiveness measures are compatible with the intent of GPRA which is to provide 
a link between measures of effectiveness and strategic planning. 

GPRA in Light of Organizational Effectiveness Analysis 

Given the recent push away from exclusive models and towards integrated models. 
Congress has attempted not to prescribe a particular model to measure organizational 
effectiveness under GPRA. The United States Senate Committee on Governmental 
Affairs report on GPRA discusses the importance of long-range strategic planning and 
proper measurement of organizational effectiveness (United States Senate, 1993:9-20). 
The report suggests that GPRA is descriptive in nature and not prescriptive. Such a 
format would give considerable discretion to organizations in developing strategic plans 
and measuring their own organizational effectiveness against them (United States Senate, 
1993:17). It should be noted, however, that the Senate report eludes to Sunnyvale, 
California as a prototype by which the government could model its strategic planning and 
organizational effectiveness measurement efforts (United States Senate, 1993:10). The 
Senate report cites several performance measures used by Sunnyvale’s local government 
in assessing its effectiveness. For example, the report cites measures such as unit costs, 
complaints per year, and phone response times (United States Senate, 1993:11). Much of 
what Sunnyvale, California measures effectiveness against are strictly output goal 
measures. By repeatedly referring to Sunnyvale, Congress may be encouraging an 
exclusive reliance on output measures, irrespective of its intent not to prescribe. 
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Government Performance and Results Act of 1993 


As mentioned in Chapter I, GPRA was enacted to increase the efficiency and 
effectiveness of the Federal government via strategic planning. Additionally, according to 
President Clinton, GPRA will “strengthen the American economy and the bonds of our 
citizenship” (Whittaker, 1995:1). The following summary outlines the multiple purposes 
of GPRA as well as its major provisions: 

GPRA’s Purpose . GPRA’s purpose can be summarized as follows: 

(1) improve the confidence of the American people in the capability of the Federal 
Government, by systematically holding Federal organizations accountable for achieving 
program results; 

(2) initiate program performance reform with a series of pilot projects in setting program 
goals, measuring program performance against those goals, and reporting publicly on their 
progress; 

(3) improve Federal program effectiveness and public accountability by promoting a new 
focus on results, service quality, and customer satisfaction; 

(4) help Federal managers improve service delivery, by requiring that they plan for 
meeting program objectives by providing them with information about program results and 
service quality; 

(5) improve congressional decisionmaking by providing more objective information on 
achieving statutory objectives, and on the relative effectiveness and efficiency of Federal 
programs and spending; and 

(6) improve internal management of the Federal government. 
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GPRA’s Maior Provisions . GPRA’s major provisions will require Federal 
organizations to: 

(a) Develop strategic plans by fiscal year 1998. 

(b) Develop annual performance plans based upon the strategic plan, setting performance 
goals beginning with fiscal year 1999. 

(c) Make performance reports annually, starting in March 2000, on actual performance 
compared to goals established in the strategic plan. 

The act also: 

(a) Allows Federal organizations to waive administrative requirements and controls under 
certain situations to provide greater managerial flexibility to get the job done in exchange 
for greater accountability. 

(b) Establishes three sets of pilot projects during the new few years. They are: 

(1) A set of pilot projects starting in fiscal year 1994, undertaken to test and 
demonstrate annual performance plans and performance reports (i.e. planning pilots). 

(2) A set of pilot projects starting in fiscal year 1995 will selected from the Federal 
organizations participating in the initial annual performance plan pilots to test and 
demonstrate the concepts involved in managerial flexibility and accountability (i.e. 
managerial flexibility and accountability pilots). 

(3) A set of pilot projects on performance budgeting (i.e. performance budgeting 
pilots). 

These major provisions and other details of GPRA are summarized in Table 2.1 below: 
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Table 2.1. GPRA Milestones and Projected Timetables (0MB, 1993:41,52) 


Wfeat's Re^pred 


■iimiiiHi 

1. Agency points of contact 
for strategic planning identified 

Sep 93 

Agencies 

2. Distribute guidance on 
strategic plan content 

Nov 93 

GMD (0MB) 

3. Agencies may submit plans that satisfy 
requirements of Section 3 of GPRA 

After 1 Oct 95 

Agencies 

4. Agencies submit initial strategic plan to 0MB and 
Congress 

By 30 Sep 97 

Agencies 

5. Three year revision and update period begins for 
agencies submitting strategic plans after 

October 1,1995 

Oct 98 

Agencies 

6. Agencies must have updated and revised initial 
strategic plan 

By 30 Sep 97 

Agencies 

7. Based on results of pilot projects, issue any 
necessary guidance on preparation of annual 
performance plans for FY 1999 

Jun97 

0MB 

8. Agencies submit annual performance plan for FY 
1999 to 0MB 

Sep 97 

Agencies 

9. 0MB authorizes agencies to use alternative forms 
of performance measurement in plans 

Oct 97 

0MB 

10. Agencies revise annual performance plans to 
reflect budgetary decisions 

Dec 97 

Agencies 

11. Agencies provide copies of complete final 
performance plan to appropriate authorizing and 
appropriation committees, and make such plans 
available to the public 

Feb 98 

Agencies 

12. At agency option, agencies revise plan to reflect 
FY 1999 budgetary decisions, and provide to 0MB, 
appropriate Congressional authorizing and 
appropriating committees, and make available to the 
public 

By 30 Sep 98 

Agencies 


What is GPRA Really About and How Will it Affect or Change Government? Despite 
its complexity, GPRA is really about change-change in the way the federal government 
does business; change in the way federal managers are held accountable for program 
results; and change in Has, focus placed on service quality and customer satisfaction. In 
effect, GPRA means that “results-oriented management is in, and spending program 
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dollars with no accountability is out” (Whittaker, 1995:3). The GPRA is intended to bring 
about a fundamental transformation in the way government operates and could be the 
catalyst for improving the overall management of the federal government (0MB, 1993:5; 
Whittaker, 1995:3). Some of the major effects GPRA is anticipated to have on the federal 
government are outlined below: 

(1) Much greater emphasis will be placed on program execution--on outputs, 
outcomes, and results. For those managers mainly concentrating on the front end (i.e. 
inputs, program definition, and policy formulation), this focus will have to be replaced 
with a focus on how programs are currently doing and what is being accomplished. This 
viewpoint is summarized by James Whittaker, who writes that “the current emphasis is 
more on process than results, and this must change,” and, “the ‘heyday’ of the input style 
of accountability appears to be over. Now it will be most important to work on the 
output, outcome, and results side of the equation” (Whittaker, 1995:4,8). Congress, in 
passing GPRA, recognized this improper focus, and they have attempted to remedy this 
problem. 

(2) No longer will strict adherence to rules and regulations be sufficient evidence of 
good management. “Put simply, process will not disappear, but the results achieved will 
be more determinate of managerial success of failure than whether one merely followed 
the rules” (0MB, 1993:6). 

(3) Organizational success will require participative management 

(4) Ineffective programs or activities will be improved, or they will be ended. Of all 
the impacts of GPRA, this is perhaps the strongest and most universal. In essence, GPRA 
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will be used as a budgeting tool in which annual appropriations are based on actual 
performance results rated against annual performance plan goals. Perhaps this explains 
why an earlier title for GPRA was the “Bang for the Buck Act” (0MB, 1993:6). The last 
of the three pilot projects under GPRA deals with what has been termed “performance 
budgeting.” Performance budgeting has been defined as “presenting the varying levels of 
performance that would result from different budgeted amounts” (0MB, 1993:5). 

Conclusion 

For the last several decades, organizational behavior researchers have devoted 
significant time to the topic of organizational effectiveness. One of the few common 
themes to come from all this research has been the realization that there is no one 
exclusive model that accurately measures organizational effectiveness in all contexts. As a 
result, the new focus in organizational effectiveness research has been towards developing 
an integrated model capable of being used in many contexts. This integrated model, 
while incorporating each of the four general approaches (i.e. output goal, multiple 
constituency, systems resource, and process) places particular emphasis on constituencies. 
In passing GPRA, Congress told the American people that government must be more 
accountable for its management actions. Additionally, GPRA, despite being more 
descriptive than prescriptive in nature, models itself after the organizational effectiveness 
model of Sunnyvale, California, a city which relies almost exclusively on output goal 
measures of effectiveness. Although current research “discourages” using one of the 
effectiveness approaches exclusively. Congress, by praising Sunnyvale, California, appears 
to be advocating, directly or indirectly, the exclusive use of the output goal approach. 
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Integrated Model of Oreanizational Effectiveness 

The integrated model depicted in Figure 2.3 below will be used to assess current SPO 
effectiveness measures. As mentioned in Chapter I, the model’s strengths are its systems 
approach, and its incorporation of GPRA’s intent to serve as the link between 
effectiveness measurement and strategic planning. 


Integrated Model of Organizational Effectiveness 


c=> 


Value-Added 
Activities 
Provided to 
Customers and 
Stakeholders 


NOTE: GPRA will provide an overall systems perspective with particular emphasis 
on outcomes, the consequences of a product or service to the customer or stakeholder. 
This consequence is in terms of value-added. Current SPO effectiveness models 
contain a limited systems perspective with particular emphasis on outputs. 


Figure 2.3. An Integrated Model for Measuring SPO Effectiveness 
Discussion of Integrated Model 

As noted in Figure 2.3, the integrated model was derived from general systems theory. 
In the context of the general systems theory of organizational effectiveness, the 
organization or program is only one element of a number of elements interacting 
interdependently. The flow of inputs and outputs is the basic starting point in describing 





Outcomes 
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the organization. In the simplest terms, the organization takes resources (inputs) from the 
larger system (environment), processes these resources, and returns them in changed form 
(output). Systems theory also stresses the organization’s connection to the larger system 
of which it’s a part. Every organization is part of an industry (a larger system), a society 
(a yet larger system), and so on. Thus, the organization can’t simply produce a product or 
service to satisfy it customers only; it must also produce products or services to satisfy 
other important components of the larger environment. In the SPO and GPRA context, 
SPOs must produce weapons systems that satisfy its immediate customer (such as ACC), 
yet it must also satisfy other stakeholders within the environment such as Congress and the 
American taxpayers (Gibson and others, 1994:34). 

The concept of the organization as a system that’s related to a larger system introduces 
the importance of feedback. As noted above, the organization depends on the 
environment not only for its inputs, but also for the acceptance of its outputs. Thus, the 
organization must develop means for adjusting to organizational demands. The means for 
adjustment are information channels that enable the organization to recognize these 
demands. As identified in the interviews conducted and from the records analysis 
performed in this study, SPOs typically employ metrics, customer surveys, and periodic 
customer meetings as both the information gathering and feedback mechanisms. 

Gibson and others illustrate effectiveness at each stage by providing the following 

example dealing with universities. A university uses resources (inputs) to teach students 

(process), to do research (output), and to provide technical information to society 

* 

(output). A university’s survival depends on its ability to attract students’ tuition and 
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taxpayers’ dollars {defined earlier in the chapter as effectiveness from the systems 
resource model’s perspective) in sufficient amounts to pay the salaries of its faculty and 
staff. If a university’s output is rejected by the larger environment so that students enroll 
elsewhere {defined earlier in the chapter as ineffectiveness from the output goal model’s 
perspective), or if a university is guilty of expending too many resources in relation to its 
output (defined earlier in the chapter as ineffectiveness from the process approach 
model’s perspective), it will cease to exist. Like a business, a university must provide the 
right output at the right price if it’s to survive (Gibson and others, 1994:35-36). This last 
sentence implies there is a difference between outputs and outcomes. This is a major 
characteristic of the integrated model depicted in Figure 2 above. The model does not 
reject the merits of the four basic models introduced earlier in the chapter(system 
resource, process approach, output goal, and multiple constituency models). Rather, the 
model integrates all four of the models and incorporates a new component called 
outcomes, defined by the model as the value added portion of the organization’s output, 
where value added means the consequences of the output to the customer/stakeholder. 
This model provides the proper orientation from which SPOs can measure effectiveness in 
a manner most favorable under the requirements of GPRA. In short, this integrated model 
stresses the major points cited in GPRA-outcomes and customers/stakeholders. 

As will be mentioned in Chapter V, this model should be tested after Congress 
mandates that SPOs implement GPRA. Since this study was a qualitative study geared at 
assessing current SPO effectiveness models and developing an integrated model of 
effectiveness, I would recommend testing the model statistically. This could be 
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accomplished by surveying SPOs’ relevant stakeholders (i.e. ACC, SAP, ANG, Congress, 
Reserves, etc.) and statistically analyzing their responses. This analysis could determine 
whether stakeholders are more satisfied under this model than in the previous, non-GPRA 
environment with the assumption that increased satisfaction results from increased 
effectiveness. 

For the purposes of this research study, we propose the following: 

1. Current effectiveness measures used within SPOs are strictly output goal 
measures; 

2. GPRA while emphasizing the need for a more “holistic” approach has not 
provided a good model to accomplish this; and 

3. An integrated model can be developed to meet both the spirit of GPRA and the 
SPOs’ need for a thorough mechanism for measuring organizational 
effectiveness 
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III . Methodology 


Overview 

This chapter introduces the methodology and research design procedures that will be 
used to answer the three propositions presented in Chapter n Literature Review: 

1. Current effectiveness measures within System Program Offices (SPOs) 
strictly output goal models; 

2. GPRA while emphasizing the need for a more “holistic” approach has not 
provided a good model to accomplish this; and 

3. An integrated model of organizational effectiveness be developed which 

will help SPOs succeed under the Government Performance and Results Act of 
1993 

The F-16 SPO will be used to assess effectiveness measures currently in use. This portion 
of the research will seek to answer proposition one above. From this assessment, an 
integrated model of organizational effectiveness will be developed. This integrated model 
will incorporate principles of each of the four general models presented in Chapter II 
Literature Review. The four general models of organizational effectiveness included: 1) 
the output goal model which focuses on output measures such as profit, costs, etc.; 2) the 
systems resource model which measures an organization’s ability to acquire limited 
resources relative to its competitors; 3) the process approach model which analyzes the 
effectiveness of an organization’s transformation processes; and 4) the multiple 
constituency model which defines effectiveness as an organization’s ability to meet its 
objectives while satisfying the interests of stakeholders (Zammuto, 1984:606). The 
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integrated model will then be applied to the F-22 SPO. This portion of the research will 
seek to answer proposition two above. 

The remainder of this chapter will introduce critical issues related to research design, 
data collection, and data analysis. 


Research Design 

Cooper and Emory state that several different design approaches exist in research “but 
unfortunately, no simple classification system defines all the variations that must be 
considered” (1995:114). Gay and Diehl propose using only two research classifications-- 
purpose and method (1992:8). Classification by purpose is defined as “the degree to 
which findings have direct business applications and the degree to which those findings are 
generalizable” (Gay and Diehl, 1992:8). Classification by method is defined as “the 
overall strategy followed in collecting and analyzing data” (Gay and Diehl, 1992:8). 

Research bv Purpose . The segregation of research by purpose yields two types-basic 

and applied research (Gay and Diehl, 1992:8). Gay and Diehl state that basic research 

takes place in a controlled environment and results in theory development/refinement; 

applied research takes theory and applies it for the purpose of answering specific research 

questions (1992:9). Emory and Cooper discuss the debate over which is “better,” and 

conclude that not all research needs to be “guided by theory and hypotheses about 

presumed relations” (1995:10). They quote Robert Dubin’s 1969 book. Theory Building : 

There is no more devastating condemnation that the self-designated theorist 
makes of the researcher than to label his work purely descriptive. There is an 
implication that associates ‘purely descriptive’ research with empty-headedness; 
the label also implies that as a bare minimum every healthy researcher has at 
least an hypothesis to test, and preferably a whole model. This is 
nonsense...Descriptive research is the stuff out of which the mind of man, the 
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theorist, develops the units that compose his theories. The very essence of 
description is to name the properties of things: you may do more, but you 
cannot do less and still have description. The more adequate the description, 
the greater is the likelihood that the units derived from the description will be 
useful in subsequent theory building. (1995:11) 

Research by Method . Gay and Diehl assert that five classifications of research by 
method can be used: 1) historical research which involves examining and explaining past 
events; 2) descriptive research which studies and reports things in their current state; 3) 
correlational research which attempts to ascertain whether two variables, events, etc. are 
related; and 4) causal and experimental research which attempts to discover causation 
between variables, events, etc. The authors present a decision tree researchers may use in 

j ^ 

choosing a research method. This decision tree is presented below in Figure 3.1. 



Figure 3.1. Decision Tree for Determining Methods of Research (Gay and Diehl, 
1992:19) 
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As will be discussed in the next sections, this study will employ applied research 
(research purpose) and descriptive research (research method). Although aspects of both 
applied and basic research may be present, the primary purpose of the study is applied 
research. The four general models of organizational effectiveness theory will be used to 
study the F-16 SPO. This portion of the study will serve two purposes. First, the study 
will reveal which models of organizational effectiveness are currently used by SPOs. 
Second, the study of the F-16 SPO will provide useful information for the development of 
the integrated model of organizational effectiveness that will be applied to the F-22 SPO. 
The F-16 portion of the study will provide answers to proposition one addressed earlier in 
the chapter and, therefore, represents applied research (through the application of the four 
general theories of organizational effectiveness). These results will provide inputs for the 
development of the integrated model to be tested in the F-22 SPO. Since no specific 
hypotheses will be tested, and because the F-22 results will be used to modify the 
integrated model, this portion of the research represents basic research. 

Data Collection 

Coomer and Moynihan summarize data collection as four critical decisions. The first 
decision is what to use as the unit of analysis. Second, the researcher must determine the 
proper methodology that will enable the researcher to answer the research 
questions/propositions. The third decision the researcher faces is selecting the proper 
sample size. Lastly, the researcher must decide on the appropriate measuring 
instrument(s) (1994:45). 
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Unit of Analysis . The researcher has limitless possibilities on which unit of analysis to 


employ. Patton states that units of analysis can range from individuals to groups, 
programs, organizations, countries, etc. (1990:166-168). Coomer and Moynihan cite 
Patton’s assertion that the bottom line in determining the: “appropriate unit of analysis is 
to decide what it is you want to be able to say something about at the end of the study” 
(1994:46). This study will attempt to ascertain organizational effectiveness models 
currently in use within SPOs. From these results, an integrated model will be developed 
and applied to the F-22 SPO, with the goal that this model will be generalizable to all 
SPOs. Given the “bottom line” approach suggested by Patton, this study will use SPOs as 
the proper unit of analysis. 

Methods . Coomer and Moynihan classify methods according to a general scheme from 
Patton’s book. Qualitative Evaluations and Research Methods . Patton classifies research 
methods as one of three types: 1) quantitative, 2) qualitative, or 3) mixed (1990:165). 
Patton distinguishes between them by stating that quantitative methods produce breadth of 
coverage of a topic area, qualitative methods provide depth of coverage of a topic area, 
and the mixed method produces both breadth and depth of coverage of a topic area 
(1990:165). This explanation suggests that quantitative methods tend to gather and test 
data from as many sources as is practicable while qualitative methods seek to completely 
investigate a limited number of areas. The research that will be conducted in this study 
will be gathered from only two SPOs, the F-16 and the F-22. Given this limited number of 
research areas, the study will be strictly qualitative. The F-16 and F-22 SPOs were 
selected based on their applicability, maturity, availability, visibility/reputation, and level of 
complexity. 
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Applicability: As was noted above, Patton states that units of analysis can include 
individuals, groups, programs, organizations, etc. However, Patton’s bottom line in 
determining the “appropriate unit of analysis is to decide what it is you want to be able to 
say something about at the end of the study.” Since this study focuses on effectiveness 
measures applicable to SPOs, SPOs are the most applicable unit of analysis. 

Maturity: The F-16 and F-22 SPOs represent the spectrum of life cycle maturity. 
Specifically, the F-16 SPO is a very mature SPO currently in the production and 
operations and support phases of the acquisition life cycle. Its processes and procedures 
are well-established, defined, and visible. On the other hand, the F-22 SPO is a relatively 
immature SPO in terms of its life cycle. It has not yet entered the production phase and 
only completed its Critical Design Review (CDR) in Apr 95. Together, the F-16 and F-22 
SPOs represent a good cross-section of the acquisition life cycle continuum along which 
all programs mature. 

Availability: Both the F-16 and F-22 SPOs are located at Wright-Patterson AFB OH, 
thus making the data available, convenient, and less costly to obtain. It is also important 
to remember that while all SPOs (i.e. organizations) are unique, they are also similarly 
organized and follow roughly the same processes and procedures. Therefore, the research 
effort will not be biased by the fact that only the F-16 and F-22 SPOs will be analyzed. In 
other words, the results obtained in this research will not be significantly altered by the 
selection of the F-16 and F-22 SPOs versus any other SPOs. 

Visibility/Reputation: Both the F-16 and F-22 SPOs are considered two of ASC’s 
more successful SPOs. Therefore, these SPOs provide the opportunity to investigate how 
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well our top SPOs measure organizational effectiveness within the context of GPRA. In 
other words, the F-16 and F-22 SPOs are the SPOs most likely to be compatible with 
GPRA at this time, and they will provide a picture of just where ASC stands in terms of its 
compatibility with GPRA. Additionally, both SPOs have employed the ASC mandated 
Integrated Product Team (EPT) organizational structure in place of the traditional 
functional/matrix structure. Although not an express purpose of this research, results 
from this study may provide some insight into the success of the new IPX organizational 
structure. 

Level of Complexity: SPOs represent the level of complexity most advantageous to 
answering the propositions addressed by this thesis. SPOs closely resemble “profit 
centers.” This does not mean profit in the traditional sense as SPOs are nonprofit. 
Government organizations. Rather, profit in this context implies that the organization is 
monitored for its effectiveness and efficiency. In other words, the organization will not 
continue to exist if their performance is poor, unlike and organization like AFMC which 
will exist indefinitely, irrespective of performance. Therefore, profit centers have a 
motivation to perform effectively and efficiently. It is this performance/survival duel that 
distinguishes a profit center from larger, “permanent” organizations. Because SPOs 
represent the highest level organization most like a profit center, they are most appropriate 
in this study in terms of level of complexity. 

It should be noted, however, that the use of a limited number of SPOs will not 

necessarily affect the generalizability of the results produced by this study. All of the 

* 

SPOs within Aeronautical Systems Center (ASC) have similar organizational structures 
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and are required to produce and report information in similar fashions. The F-16 and the 
F-22 SPOs are organized in a manner similar to all SPOs, and the results generated by 
studying only these two can be applied to all SPOs. This degree of 
applicability/generalizability may vary from SPO to SPO, however, the results of this study 
will provide useful information to all SPOs. 

Instruments . Data collection for qualitative methods can be accomplished in a number 
of ways. Emory and Cooper cite nine approaches to data collection in qualitative research 
(1995:118-119). These methods include: 

1. In-depth-interviewing 

2. Participant observation 

3. Films, photographs, and videotape 

4. Projective techniques and psychological testing 

5. Case studies 

6. Street ethnography 

7. Elite interviewing 

8. Document analysis 

9. Proxemics and kinesics 

This study will focus on three of the nine approaches to data collection cited by Emory 

and Cooper. Specifically, this study will use in-depth-inter\'iewing, document analysis and 

direct observation. Patton describes these data collection techniques as follows: 

data from interviews consist of direct quotations from people about their 
experiences, opinions, feelings, and knowledge. The data from observations 
consist of detailed descriptions of people’s activities, behaviors, actions, and 
the full range of interpersonal interactions and organizational processes that 
are part of observable human experience. Document analysis in the qualitative 
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inquiry yields excerpts, quotations, or entire passages from organizational, 
clinical, or program records; memoranda and correspondence; official 
publications and reports; personal diaries; and open-ended written responses 
to questionnaires and surveys. (1990:10) 

This study will conduct interviews with numerous people within both the F-16 and F-22 
SPOs via structured interviews. The interview questions and their purposes are outlined 
below in Table 3.1: 


Table 3.1. Interview Question and Purpose Matrix 




Question#!: How do you measure your 
ability to meet your customers’ needs? 

To assess the level of customer and 
stakeholder awareness and involvement 
within SPOs; To reveal the types of 
measures used by SPOs in measuring 
customer satisfaction. 

Question #2: What is your definition of an 
effective program? What do you believe is 
the SPO Director’s definition? 

To identify if SPOs view effectiveness in 
terms of inputs, processes, outputs, or 
outcomes; To identify what lower level 
personnel perceived to be the SPO 

Director’s definition. Was it input 
oriented? Process oriented? Output 
oriented? Outcome oriented? 

Question #3: Given the way you currently 
measure effectiveness, do you consider this 
to be proactive or reactive? 

To determine whether current effectiveness 
measures were perceived to be proactive or 
reactive. 

Question #4: Given the way you currently 
measure effectiveness, can you think of any 
ways to improve the measurement process? 

To identify any potential areas which could 
be factored into the integrated 
organizational effectiveness model 
produced by this study. 


Additionally, document analysis will be performed by looking at minutes from program 
reviews, unit self-assessments, metrics reports, integrated product team (IPT) meeting 
minutes, etc.. The main records to be analyzed are outlined below in Table 3.2: 
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Table 3.2. Matrix of Records Reviewed 


1 


IHHHII 


Unit Self- 

Assessments (USAs) 

Annually 

SPO Director 

Annual SPO assessment 
of seven areas to 
include: 1) leadership, 

2) Information & 
Analysis, 3) Strategic 
Planning, 4) Process 
Management, 5) 

Quality, 6) Operational 
Results, and 7) 

Customer Satisfaction 

Executive Program 
Management 

Reviews (EPMRs) 

Quarterly 

Mj|H 

Provide SPO Director a 
broad based review of 
SPO performance 

Requirements and 
Planning Councils 
(R&PCs) 

Quarterly 

Top SPO Managers 
and main SPO 
customers to include 
ACC. ANG, 

Reserves, etc. 

Get SPO management 
together with the 
definer of requirements, 
ACC, to plan how SPO 
will pursue ACC’s 
long-term requirements 

Kenne Interviews 

Unknown 

ASC/CV meets with 
Air Mobility 
Command (AMC), 
ACC, Air Education 
and Training 
Command (AETC), 
and other major 
customers 

Gives ASC customers 
opportunity to highlight 
their areas of 
importance and concern 

ASC/CC Mandated 
Metrics 

Monthly 

SPO Personnel 

Give ASC/CC insight 
into each of 22 
programs’ current 
performance 


The records review serves the following purposes: 

1. Validates SPOs track and monitor program performance and effectiveness 

2. Assesses the existence and extent of any SP'O strategic planning being accomplished 

3. To illustrate top-level performance indicators/metrics used within SPOs 
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Lastly, direct observation of SPO members activities will be conducted during on-site 
visits. By combining these three approaches, the study will capitalize on the principle of 
“triangulation” (Patton, 1990:187) which is discussed in the next section. 


Triangulation 

In the area of research, triangulation is defined as “the combination of methodologies in 

the study of the same phenomena or programs” (Patton, 1990:187). This can mean using 

several kinds of methods or data, including using both quantitative and qualitative 

approaches. Additionally, mixes of measurement, design, and analysis are possible 

(Patton, 1990:189). The term triangulation is borrowed from land surveying and basic 

geometry. Knowing a single landmark only locates you somewhere along a line in a 

direction from the landmark, whereas with two landmarks you can take bearings in two 

directions and locate yourself at their intersection (Fielding and Fielding, 1986:23). 

Denzin adds that the logic of triangulation is based on the premise that: 

no single method ever adequately solves the problem of rival causal 
factors...Because each method reveals different aspects of empirical 
reality, multiple methods of observations must be employed. This is 
termed triangulation. I now offer as a final methodological rule the 
principle that multiple methods should be used in every investigation. 
(Denzin, 1978:28) 

This passage reveals that triangulation provides increased breadth of coverage of a 

particular phenomena or program. Other important strengths of triangulation are its 

ability to reduce risk and increase validity. Patton writes: 

Studies that use only one method are more vulnerable to errors linked 
to that particular method (e.g., loaded interview questions, biased or 
untrue responses) than studies that use multiple methods in which 
different types of data provide cross-data validity checks (emphasis 
added). (Patton, 1990:188) 
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Denzin has identified four basic types of triangulation in research: (1) data triangulation- 
-the use of a variety of data sources in a study; (2) investigator triangulation—the use of 
several different researchers or evaluators; (3) theory triangulation-the use of multiple 
perspectives or theories to interpret a single set of data; and (4) methodological 
triangulation—the use of multiple methods to study a single phenomena, program, or 
problem (Denzin, 1978:28). 

This thesis will employ two of the four types of triangulation proposed by Denzin. 
Specifically, this study will use aspects of data and theory triangulation. Data 
triangulation, or the use of a variety of data sources in a study, will be accomplished by 
using several data sources during the study. These data sources include interviews, 
records analysis, unit self assessments, and direct observation. Theory triangulation, or 
the use of multiple perspectives or theories to interpret a single set of data will be 
accomplished by applying four traditional models of organizational effectiveness to the 
data gathered via interviews, records analysis, etc. These models are the output goal, 
multiple constituency, systems resource, and process approach models that were presented 
in Chapter 2. These models will act as filters through which all research data will be 
“strained” to evaluate the data’s orientation to each model. 

The use of the triangulation technique will produce overlap and redundancy within the 
results. These areas of overlap produced by triangulation will be a main focus of the 
research and will be the main source of information used to modify the integrated model of 
organizational effectiveness developed in Chapter II Literature Review. 
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IV . Data Analysis and Findings 


Overview 

Chapter II concluded by presenting two propositions to be addressed by this study. 
These propositions were identified as follows: 

1. Current effectiveness measures used within SPOs are strictly output goal 
measures; 

2. GPRA while emphasizing the need for a more “holistic” approach has not 
provided a good model to accomplish this; and 

3. An integrated model can be developed to meet both the spirit of GPRA and the 
SPOs’ need for a thorough mechanism for measuring organizational effectiveness 

As noted in Chapter III, triangulation was defined as the combination of methodologies in 
the same study. This study capitalized on two types of triangulation—data triangulation 
and theory triangulation. Data triangulation, was accomplished through interviews, 
records analysis. Unit Self Assessments (USAs), and direct observation. Theory 
triangulation, was accomplished by applying the four traditional models of organizational 
effectiveness introduced in Chapter 11 to the data gathered from interviews, records 
analysis, etc. Recall that the strength of triangulation is its ability to identify common 
patterns within data gathered fi'om various sources. The overlap produced by 
triangulation resulted in increased breadth of coverage, reduced risk, and increased 
validity. 

This chapter will present and analyze the data g'athered during this study. First, the data 
gathered from interviews with F-16 and F-22 SPO personnel will be presented and 
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analyzed. This will be accomplished by addressing each interview question individually in 
terms of the results/fmdings they produced. 

After presenting the interview results, data collected from SPO records are be 
presented and analyzed. This data was gathered from records that included Executive 
Program Management Reviews (EPMRs), Unit Self Assessments (USAs), and metrics 
reports. The records data analysis wiU be accomplished in the same manner as was used in 
analyzing individual interview questions (i.e. purpose, results/findings, and 
analysis/conclusions). Lastly, based on the data analysis and conclusions reached, the 
integrated model for measuring organizational effectiveness developed in light of GPRA 
will be used to depict the current state of SPO effectiveness measures. 

Interview Questions 

Interview Question #1 : How do you measure your ability to meet your customers’ 

needs? 

Results/Findings : 

The interviewees named several methods used to measure their ability to meet their 
customers’ needs. These methods were broken down into instmments and forums. 
Instruments used included surveys, metrics, and management reviews. Forums in which 
SPOs met with their customers included EPMRs, R&PCs, and strategic planning offsites. 
Actual measurements were separated into periodicAong-term (surveys, EPMRs, R&PCs, 
and offsites) and day-to-day (metrics). 

Interview Question #2 What is your definition of an effective program? What do 

you believe is the SPO Director’s definition? 
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Results/Findings : Most of the respondents answered that effective programs are programs 
that meet the customer’s stated requirements. This indicated that most of the respondents 
associated effectiveness with customer satisfaction. This viewpoint, however, was not 
shared by all respondents. Several of the respondents cited timeliness and cost savings as 
the central determinants of an effective program. For example, one interviewee kated that 
an effective program is one that sticks as closely as possible to original targets in terms of 
Cost/Schedule Control Systems Criteria (C/SCSC). Another interviewee responded that 
Contract Administrative Lead Time (CALT), a measure of timeliness, was the most 
important and only real measure of effectiveness within the contracting domain. These 
last two examples indicate that these interviewees associate effectiveness with strictly 
output measures such as timeliness and cost savings. 

Regarding the SPO Director’s definition of an effective program, interviewees 
regarded this as a problem in most instances. Several F-16 interviewees stated that there 
have been three SPO Directors since 1993. The result of such turnover has been an 
unstable focus for lower level employees, as they are unsure of which direction the SPO is 
headed. Additionally, several SPO employees and interviewees mentioned a previous 
ASC Commander mandate that all SPOs report a limited set of common, standardized 
metrics to ASC. These limited, standardized metrics included eighteen metrics dealing 
with cost, schedule, and performance only. The eighteen metrics as applied to the F-16 
SPO are listed below in Table 4.1: 
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Table 4.1. ASC/CC Mandated Metrics 


U. Standardized Cost Metrics 

MMSi— 

Stsindacdked Feifoimanoe 
Metrics 

Planned v. Actual Obligations 

Contractor Production 

Profile 

Mission Capability (F-16 

A/B Models) 

Contract Cost Variance 

Schedule Variance 

Mission Capability (USAF 
F-16 C/D Models) 

Required v. Budget 
(Development) 

Key Milestones (Block 50 
OFP Tape 4) 

Mission Capability (ANG 
F-16 C/D Models) 

Required v. Budget 
(Production) 

Key Milestones (F-16 1994 
Buy) 

Mission Capability (AFR 
F-16 C/D Models) 

Required v. Budget (O&M) 

Key Milestones (Modified 
Modular Mission Computer) 


Program Cost Disconnect 

Key Milestones (Digital 
Terrain System) 

Technical Performance 
Parameters 


The employees and interviewees added that the then ASC Commander did not 
communicate with the SPOs on which metrics would be used or why the metrics chosen 
were deemed so important. They also added that the metrics chosen were not used by 
SPO managers to manage their daily responsibilities. 

Interview Question #3 Given the way you currently measure effectiveness, do you 

consider this to be proactive or reactive? 

Results/Findings : Several of the respondents cited this area as one of big concern. They 
stated that many of the metrics employed were too short term to be beneficial to daily 
planning and management. Additionally, they stated that many of the measures were 
received after the completion of the activity of interest. That is, many of the measures 
were strictly output measures which could not help planning, because by their very nature, 
these measures were after the fact. All the respondents did state, however, that many of 
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their SPO’s metrics provide timely information which helps daily planning and 
management. 

Several of the respondents also mentioned that SPOs, being Government 
organizations, change so frequently that they tend to be/are forced to be reactive and 
oriented to the short-term. That is, the respondents felt that many Government ' 
organizations are too constrained to engage in proactive, long-range planning. For 
example, personnel in contracting felt that their specific areas are too volatile to be 
suitable for stable, long-range planning. This sentiment has been echoed by Congress, 
which in passing GPRA noted that “government management tends to be driven by the 
constraints of the organization, not the tasks of the organization” (United States Senate, 
1993:3). 

Interview Question #4 Given the way you currently measure effectiveness, can you 

think of any ways to improve the measurement process? 

Results/Findings : The interviewees had several suggestions on how to improve the 

measurement process. These suggestions are highlighted below: 

1) One interviewee was bewildered as to why SPOs were asked to reorganize before 
they had first defined all of their processes. In the interviewees eyes, SPOs were forced to 
implement the IPT organizational structure before it had defined all of its processes and 
without any statistical validation that the IPT structure was a better organizational 
structure. The interviewee argued that more time should be spent on defining, 
understanding, and publicizing SPO processes. This would have a greater impact on what 
is measured and how it is measured than does a reorganization. 
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2) Several interviewees commented on the use of qualitative versus quantitative 
measures of effectiveness. One interviewee believed that long-range goals (i.e. strategy) 
should include abstract items, whereas short-range goals (i.e. tactical objectives) should be 
quantifiable. In short, these interviewees believed that qualitative measures drive 
quantitative measures. The interviewees argue that since too many people believe only 
quantifiable areas should be measured, we tend to manage using short-range tactical 
objectives and no overarching long-range strategic umbrella. In other words, strategy is 
neglected because it tends to be long-range and abstract. As a result, strategic planning 
tends to be very weak. 

3) One interviewee noted that many different organizations interact with SPOs and 
that each can serve as a customer to some or all of these organizations. Despite the 
existence of many customers such as the contractor. Air Combat Command (ACC), the 
Air National Guard (ANG), the Secretary of the Air Force’s Staff (SAF), Congress, etc., 
no formal strategic planning is accomplished jointly with all customers. For example, the 
F-16 strategic planning document was developed by the SPO and the contractor, 
Lxickheed Fort Worth Company (LFWC) at an offsite in Jan 95. Yet ACC, the ultimate 
end user of the F-16, were not part of the offsite. The interviewee noted that all players 
are brought together at quarterly Requirements and Planning Councils (R&PC), but that 
currently formal strategic planning was not an activity included in the meeting. The 
interviewee suggested that formal strategic planning be accomplished at these meeting 
arguing that goals could be established for all customers. 
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Records Analysis 
Results/Findings : 

Unit Self-Assessments (USAs\ USAs provided the most extensive documentation and 
insight into how SPOs track and monitor performance, to what degree strategic planning 
is being used, and what types of performance indicators are used within SPOs. As 
indicated in Table 3.2, each SPO’s USA addresses seven areas, including strategic 
planning and metrics in Sections 3 and 6, respectively. Each SPO’s USA indicated that 
strategic planning is taking place. For example, the F-22 strategic plan is called the F-22 
Team Plan. The plan was jointly developed by senior management from both the SPO and 
contractors, as well as representatives from major customers such as ACC and Secretary 
of the Air Force (SAF) staff. The plan is reviewed annually at an executive offsite and 
updated accordingly. The annual offsite produces “goals” and “focus areas” to support 
the overall strategic plan. Goals capture the long-term objectives (one to six years) and 
the focus areas delineate the short-term objectives (one year or less). 

The F-16 SPO’s USA also showed evidence of strategic planning. However, the F- 
16’s strategic planning process is relatively new. For example, the F-16 USA states that a 
new initiative was started in Jul 94 to develop a strategic planning process the links 
existing processes into an Integrated Strategic Master Plan (ISMP). From there, a senior 
management offsite was accomplished in Jan 95. Additionally, the F-16 SPO has 
developed nineteen “implementation plans” which are analogous to the F-22’s goals and 
focus areas concept. 
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The USAs also validated that SPOs track program performance and have top-level 
metrics for measuring performance. Section 6 of both SPO’s USAs depicted numerous 
metrics covering many program areas. Although these metrics covered many areas, there 
was a substantial amount of attention devoted to cost, schedule, and performance metrics. 

Executive Program Management Reviews (EPMRs) . EPMRs also validated that SPOs 
do track and monitor program performance. A key indicator of senior management’s 
focus was revealed in the area of action items. Action items are areas needing action or 
attention as determined by top management within the SPO. For example, the F-16 
EPMRs monitor thirteen different functional areas including contract management and 
financial management, the traditional owners of cost, schedule, and performance metrics. 
Of the thirteen action items in EPMR #9, four were in the area of contract and financial 
management. Of the eighteen action items in EPMR #8, five were in the area of contract 
and financial management. 

The EPMRs also established that strategic planning is becoming more important to top 
management. In the F-16, both EPMRs 8 and 9 included a briefing on the current status 
of strategic planning within the SPO. 

Kenne Interviews . Brigadier General Kenne’s interviews with AMC, ACC, AETC, 
HQ AFR, and the Air Force Special Operations Command (AFSOC) revealed three areas 
were deemed important from the customer’s perspective. The three areas were 
communications, responsiveness, and strategic planning. Of particular interest was the 
emphasis the customers are placing on strategic planning. 
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In addition. Brigadier General Kenne also had each command prioritize the more 
traditional indicators of cost, schedule, and performance. The commands voted that, even 
in times of severe fiscal constraint, performance is their number one priority. Cost and 
Schedule were rated relatively equal, but both were below performance. 

The commands were also asked to identify any items they considered to be better 
indicators of system performance than acquisition cost, schedule, and performance. The 
commands identified three additional indicators of effectiveness. They were life cycle 
cost, standardization and simplification of parts across different weapon systems, and the 
size of the mobility footprint. 


Integrated Model of Organizational Effectiveness as it is Currently 
Implemented in SPOs 




Products 

or 

Sendees 


0 


<=> 



<=1 




NOTE: GPRA will provide an overall systems perspective with particular emphasis 
on outcomes, the consequences of a product or service to the customer or stakeholder. 
This consequence is in terms of value-added. Current SPO effectiveness models 
contain a limited systems perspective with particular emphasis on outputs. 


Figure 4.1. Current State of SPO Effectiveness Measurement Process in Light of 
Integrated Model of Organizational Effectiveness 
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Figure 4.1 above illustrates the current situation within the F-16 SPO and the F-22 
SPO. As shown in the interviews and records reviews, the data suggest that SPOs have 
placed a heavy emphasis on short-term, efficiency or output measures of effectiveness. 
This is not to say that SPOs have totally excluded outcome measures or ignore customer 
feedback. However, SPOs appear to give much more weight to output measures such as 
Cost/Schedule Control Systems Criteria (C/SCSC). Unit Self-Assessments did provide 
evidence that SPOs seek to use outcome measures. By including their main customer, 
ACC, the F-22 SPO successfully incorporated the outcomes and environment aspects of 
the integrated model. However, this did not change the perception of most interviewees 
who concluded the SPO was much more focused on short-term, output measures. This 
indicates a possible disconnect between top management (preparers of USAs) and lower 
level employees (interviewees) within the SPO. Such disconnects have been cited by 
Schmenner and Vollmann in an article discussing performance measures. The authors’ 
research used a statistical methodology that compared how employees and top 
management ranked various performance measures in terms of their overall impact on 
organizational success. The results suggested a big difference between what each group 
thought was important to organizational success. These differences between groups were 
termed by the authors as “gaps” and “false alarms.” A gap was defined as “failing to use 
the right measure, so that something important stays neglected.” False alarms were 
defined as “using the wrong measure to motivate managers so that they spend time 
improving something that has few positive, and perhaps many harmful, consequences” 
(Schmenner and Vollmann, 1993:58). SPOs should be very wary of such disconnects. 
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V. Conclusions and Recommendations 

Overview 

This chapter presents the conclusions drawn from the data presented in Chapter FV and 
addresses the individual research propositions. Limitations to this thesis as well as 
recommendations for future research are also provided. 

Conclusions 

The conclusions based on individual interviews are presented below: 

Interview Question #1: How do you measure your ability to meet your customers’ needs? 

The results from this question indicated that SPOs interact with customers via many 

different mediums to include surveys, EPMRs, R&PCs, etc. Despite the many 

opportunities to interact with customers to analyze and assess customer satisfaction, SPOs 

have relied on metrics to measure their ability to meet customers’ needs. By itself, this is 

not wrong, however, it does appear that SPOs can capitalize on the face-to-fact forums 

such as EPMRs and R&PCs. Perhaps strategic planning or team building exercises could 

be added as formal requirements at every EPMR and/or R&PC. Fact-to-face forums 

provide SPOs with the best opportunity to discover and measure customer needs. By 

capitalizing on the face-to-face forums, SPOs could increase the probability that they are 

providing value-added outcomes instead of just measuring pure outputs. 

Interview Question #2: What is your definition of an effective program? What do you 
believe is the SPO Director’s definition? 

Results from this question indicate that SPO personnel covering the entire chain-of- 
command define an effective program as one which provides good output results such as 
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timeliness and cost savings. Given that a former ASC Commander dictated that effective 

programs can be defined in terms of seventeen cost, schedule, and performance metrics, 

one can conclude that this is the prevailing definition of program effectiveness. At best, 

the results may indicate a serious disconnect between top management and lower level 

employees. At worst, the results may indicate that SPOs are relatively unaware of the 

outcomes or value-added consequences of products they provide to their customers. It is 

important to recognize that cost, schedule, and performance problems are at the core of 

Congressional decisions to fund or kill programs. However, it is equally important to note 

that such a preoccupation may be to the detriment of outcomes, and therefore customers. 

Interview Question #3: Given the way you currently measure effectiveness, do you 

consider this to be proactive or reactive? 

Change and volatility pervade in nearly all organizations. The majority of interviewees 
concluded that Government organizations, especially SPOs, are too volatile to engage in 
proactive measuring and strategic planning. This volatility stems from a changing political 
scene, changes in funding, changes in SPO leadership, etc. and has always been present. 
The interviewees argued that volatility causes more emphasis on the immediate future. 
Consequently, managers measure short-term effectiveness with little concern for longer 
range goals and strategies. As mentioned in Chapter IV, Congress recognized this when it 
stated “government management tends to be driven by the constraints of the organization, 
not the tasks of the organization.” GPRA will force top managers to place more emphasis 
on longer range goals and strategies. However, is it possible that Congress has potentially 
underestimated the impacts of higher levels of change and volatility to the point that 
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GPRA, as written, promotes short-term, output measures of effectiveness, measures 

which have a very minimal impact on long-range strategic planning and effectiveness? 

Interview Question #4: Given the way you currently measure effectiveness, can you think 
of any ways to improve the measurement process? 

The recommendations provided by the interviewees could be reduced to three general 

recommendations listed below: 

1. Place more emphasis on defining and understanding organizational processes 

2. Promote greater use of qualitative measures of effectiveness 

3. Increase the amount of formal strategic planning 

Recommendation #1 suggests that to improve how SPOs measure performance they must 
understand their business. Such an understanding of the business would increase their 
chance of knowing just what it is that the customer wants Recommendation #2 stresses 
the belief that'many quantitative measures tend to be geared to the short-term. Such a 
focus is to the detriment of the long-term and therefore, strategic planning. 
Recommendation #3 implies that formal strategic planning should be incorporated into 
high-level meetings such as EPMRs and R&PCs, meetings in which the SPO interacts 
face-to-face with many important customers. Perhaps this would increase the chances of 
SPOs developing more outcome measures of effectiveness. 

The conclusions based on records analysis are presented below: 

1. There appears to be a possible disconnect between top SPO management and top 
ASC management. The records review revealed, as was the case with individual 
interviews, that the ASC mandated metrics are not necessarily the same metrics used by 
top SPO managers to run the program. Not only did this indicate a possible misalignment 
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of focus between ASC management and SPO management, but it indicated that ASC 
management defines effectiveness in terms of cost, schedule, and performance measures, 
measures which have traditionally been output measures of effectiveness. 

2. Brigadier General Kenne’s interviews with several major commands indicated that 
more emphasis is being given to strategic planning. Additionally, the commands* rankings 
of performance ahead of cost and schedule, even amid severe fiscal constraints, suggests 
commands place more emphasis on outcomes than outputs. System performance is more 
likely to result in effectiveness than cost or schedule performance. In other words, system 
performance is more likely to be a valuable outcome measure whereas cost and schedule 
performance are more likely to be output measures. Perhaps if customers continue to 
stress performance, any potential SPO reliance on strictly output measures as measures of 
effectiveness will be decreased. 

EPMRs tracking of action items suggests heavy emphasis is placed on cost, schedule, 
and performance areas. As mentioned earlier, cost, schedule, and performance problems 
can be “show stoppers” in Congress, but a total fixation on those areas tends to be at the 
expense of keeping a good customer focus. 

Recommendations 

1. In order to comply with GPRA, SPOs must place more emphasis on other areas of 
the integrated model besides outputs. Specifically, SPOs must increase efforts to 
incorporate the outcomes and environmental characteristics of the model. For example, 
recommendation #3 from the interviewees was for SPOs to engage in more formal 
strategic planning at face-to-face forums such as EPMRs and R&PCs. The interviewees 
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believed this would increase the likelihood of SPOs developing more outcome measures of 
effectiveness. Such forums provide an excellent opportunity for SPOs to receive feedback 
from customers. In short, more formal strategic planning with customers would help 
SPOs internalize the outcome characteristic of the model, and face-to-face forums would 
allow SPOs to receive feedback and therefore cover the environmental characteristic of 
the model. Such an increased emphasis is depicted in Figure 5.1 below: 


Integrated Model of Organizational Effectiveness 




Processes 



Outputs 






Products 

or 

Services 


Value-Added 
Activities 
Provided to 
Customers and 
Stakeholders 




NOTE: GPRA will provide an overall systems perspective with particular emphasis 
on outcomes, the consequences of a product or service to the customer or stakeholder. 
This consequence is in terms of value-added. Current SPO effectiveness models 
contain a limited systems perspective with particular emphasis on outputs. 


Figure 5.1. How SPOs Can Improve Their Current Effectiveness Measurement Processes 


2. As mentioned in Schmenner and VoUmann’s article on performance measures, many 
organizational performance measures are plagued by gaps and false alarms. To avoid 
these pitfalls, it may be useful for SPOs to periodically examine their performance 
measures in much the same way the article did. Perhaps senior leadership could let the 
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people within the SPO rank the performance measures currently being used within the 
SPO. Such a ranking would be beneficial to senior leadership for several reasons. First, 
the rankings would reveal which measures within the SPO are actually viewed as 
important to day-to-day activities. This perspective would be the perspective of middle 
managers and lower level employees, the people that perform the day-to-day management 
of the program. Second, it would allow middle managers to voice their opinions on what 
performance measures should be added to improve the program. In short, allowing lower 
level personnel to rank/evaluate the performance measures used in the SPO would ensure 
that senior leadership and middle managers manage areas jointly determined to be 
important. If senior management and middle management are not management from the 
same basket of important effectiveness measures, much inefficiency and loss of direction 
will result as both parties pursue different areas. As was mentioned in Chapter IV, a 
recent ASC Commander mandated that SPOs measure and report a common set of 
metrics to his office. Interviews conducted during this study revealed that the vast 
majority of middle managers did not use the common set of metrics to manage the daily 
activities within their area of responsibility. 

3. All SPOs must understand that GPRA places even more importance on Congress as an 
important stakeholder. Specifically, Congress intends to use performance budgeting to 
appropriate monies to SPOs based on their performance. Of course, SPOs are a unique 
entity that due to its very nature may get “special” treatment. That is, many SPOs exist to 
fill a void in our nation’s defense capability. It is unlikely that such critical, sole source 
SPOs will lose all funding if their performance is unsatisfactory. The A-12 is an example 
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of a program that was canceled due to poor performance. However, I would argue that if 
the A-12 was the only system capable of filling the Navy’s void, it most likely would not 
have been canceled. My point is that performance budgeting will impact SPOs, however, 
because many SPOs exist to fill a unique requirement, the limits of performance budgeting 
may be different than from other, “normal” federal agencies and activities. Perhaps this 
issue can be studied after SPOs have been subjected to performance budgeting for several 
years. 

4. Awareness of GPRA must be increased. The only individuals I spoke with during this 
study who had even heard of GPRA were in the ASC Quality office. Aside from that 
office, no one I spoke with had ever heard of it. If GPRA will have the impact it purports 
to, middle managers and up need to be aware of its impacts on performance measures and 
strategic planning as soon as possible. The more educated SPO management becomes 
early on, the higher their chances for success when GPRA is mandated in their program 
office. 

5. To comply with GPRA, SPOs need to ensure that each item within their strategic plan 
can be measured and assessed prior to the output stage. It is also important to note that 
measurement does not mandate a quantitative solution. Many qualitative measures (i.e. 
surveys, interviews, etc.) can also explain whether or not an organization is performing 
effectively. Numbers by themselves may be somewhat misleading. GPRA allows for 
qualitative goals so long as they can be measured. 

6. I would argue that the F-22 SPO, as it currently operates, is very close to meeting the 
requirements of GPRA. A review of the F-22’s Unit Self Assessment (USA) indicated 
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that the SPO has already built a solid foundation for success under GPRA. Specifically, 
the F-22 SPO implemented strategic planning from the start and manages according to 
their strategic plan. Additionally, the SPO conducts regular “what if’ analyses that look at 
the potential impacts of various funding scenarios. This essentially amounts to an exercise 
in performance budgeting. In other words, I believe the F-22 SPO in many ways, has 
already implemented GPRA into their daily operations, and many SPOs could learn from 
the example the SPO has set. 

7. This research to a large degree represents naturalistic inquiry, defined by Patton as a 

“discovery oriented approach that minimizes investigator manipulation of the study setting 

and places no prior constraints on what the outcomes of the research will be.” Patton 

goes on to say that naturalistic inquiry can be considered the opposite of experimental 

design where the investigator attempts to control study conditions by manipulating or 

holding constant external influences (Patton, 1990:41). From this explanation, we can 

consider naturalistic inquiry to be relatively equivalent to qualitative study. When 

speaking of generalizability and naturalistic inquiry, Patton writes: 

The evaluator should do what he can to establish the generalizability of 
his findings...Often naturalistic inquiry can establish at least the ‘limiting 
cases’ relevant to a given situation. But in the spirit of naturalistic 
inquiry he should regard each possible generalization only as a working 
hypothesis, to be tested again in the next encounter and again in the 
encounter after that. For the naturalistic inquiry evaluator, premature 
closure is a cardinal sin, and tolerance of ambiguity a virtue. 

(Patton, 1990:488) 

Therefore, the generalizability of my findings and also my model must be 
tested/revalidated again and again. My findings should be regarded as working hypotheses 
to be tested in future thesis efforts. 
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Potential Follow-on Studies 


Based on the limited generalizability of my findings, I would suggest the following 
potential follow-on research efforts: 

1. The model developed in this study should be applied to SPOs in future research to 
validate its ability to predict effectiveness. Additionally, the model should be refined as 
new findings are produced and tested in SPOs after the implementation of GPRA. By 
testing a refined model after GPRA implementation, potential predictive capabilities of the 
model can be realized. Additionally, post-GPRA implementation testing of the model 
would provide the strongest case for its generalizability to other SPOs and, potentially, to 
other federal agencies within limits. 

2. Any future follow-on research conducted should include the F-16 SPO. Since the 
SPO only began “formal” strategic planning in Jan 1995, much of the data gathered needs 
more time to stabilize and mature. 

Limitations of Study 

Based upon the methodology used in this thesis, several limitations must be 
recognized. These limitations restrict the level of generalizability my findings may be 
given. These limitations include the following: 

1. The data used in this study were strictly cross-sectional data. That is, the data gathered 
represented only a point in time. Future research may seek to gather time series data, or 
data gathered over time. 
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2. The units of analysis in this thesis were SPOs. As mentioned in Chapter III, gathering 
data from levels above SPOs would have been too complex. As such, the ability to 
generalize my findings to other types of organizations is limited. The integrated model 
itself can be applied to any organization, however, the findings of this study deal stricdy 
with SPOs. To generalize my findings to other types of organizations will require testing 
within those organizations. 

3. The data were collected from two SPOs, both located at Wright-Patterson AFB OH. 
This represents a geographic limitation as Wright-Patterson AFB may follow slightly 
different procedures than SPOs located at Eglin AFB FL, Lx>s Angeles AFB CA, or 
Hanscom AFB MA. Future studies may benefit from collecting data from two or more 
geographic locations. 
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